Bringing robots and people together

What is this project about?

The human body project is an attempt to enable computers or computer controlled robots to understand what humans are and what they're up to.  For almost all of its history the digital computer has been a supremely stupid machine - typically a box with a keyboard, mouse and screen as its only way of sensing and responding to instructions.  Computer programs represent some of the most complex structures devised by the human mind, but in spite of all their apparent sophistication the typical messages which they come out with, such as " illegal operation", reveal just how socially inept they are.

At the beginning of the 21st century after several decades of Moore's law the processing power and wide availability of computers is starting to enable them to interact with people in a new way, using vision and speech interfaces rather than the clumsy keyboard bashing of the past.  Perhaps before long computers will be able to "see" the people around them and respond to them in a more humanistic way rather than just treating them as usernames and passwords.

I've spend all my working life as a software engineer, a significant proportion of which has been in the field of industrial automation.  I've been involved with programming robotic machines of various kinds to perform complicated types of movement, and the amount of time and effort involved in such projects is often exorbitant.  I've often thought it would be nice just to have me gesture the kind of motion which I want and then just say to the computer "do this".  Wouldn't it be nice if computers could learn more like people from observation rather than by being directly programmed by an expert.

The human body project is my attempt to do just that, using a webcam and a desktop PC and with the help of a small robot named Rodney.  Using the webcam which constitutes one of the robots eyes he can be aware of my presence in the room, make judgements about where I am and how far I am away, what my arms are doing and what I'm looking at, and perhaps most importantly of all be able to identify me uniquely by sight.

Mr Plong

Mr Plong is a humanoid avatar who will attempt to imitate the facial and body postures observed via the webcam.

The main systems being developed in the human body project are:

Face Recognition - the ability to learn people's faces and identify them uniquely.

Gaze Direction - what direction is the person looking?

Facial expressions - finding the positions of facial features which may be indicative of various emotional states.

Body Posture Recognition - detecting the upper half of the body, body size, colour of clothes, arm positions and so on.

Personal interaction classification - classifying the ways in which each individual  interacts with the system

Source Code

This is a non-commercial open source project under the GNU General Public Licence.  It is written using MS Visual C++ 6 and Visual Studio 2003, and utilises the face detector from the Intel OpenCV image processing library.  The project also currently uses the superb EDISON segmentation algorithm developed by Chris Christoudias and Bogdan Georgescu.  

You can download the latest source code from the SourceForge site.

The code is organised in the following directories:

API Documentation

Further details of how to call the various functions within the C++ DLL can be found here. 

Applications of the Human Body

Applications which use the Human Body Project are:

The Doorbell of Doom   Detect and announce people as they enter a room.

Project Notes

This project aims to detect human upper body postures and facial expressions.  It consists of a DLL written using Microsoft Visual C++ 6 and a front end application written using visual studio 2003 (mainly in vb.net).  All code is brought to you courtesy of the GNU General Public Licence.

To try to make the program a little easier to understand I have included XML comments compatible with NDoc.  Program documentation may be produced by installing VBCommenter, building the VB.net project then using NDoc to create a document in the format of your choice.

There are a number of things which need improvement.

1.  Detection of the eyes and left/right sides of the face is sometimes unreliable.  At a minimum some consistency checking is needed to reject configurations of features which obviously aren't a face.  Poor detection of the sides of the face leads to errors in the estimation of gaze direction.

2.  There are some problems which are probably just the result of my unfamiliarity with .NET.  I've found it very difficult to pass a fixed length array of bytes (typically a bitmap) from the front end VB.net program to the back end DLL (or vice versa).  After struggling to do this properly I eventually gave up, and the present system passes one pixel at a time to the DLL which is very inefficient.  There must be a better way to do it than this.  If anyone knows what it is please contact me at fuzzgun@btinternet.com

3.  Detection of arms presently assumes that the colour of your arms is the same as the colour of your body.  So, if you're wearing a T-shirt it doesn't work, or becomes very unreliable.  This needs improvement.  With the system as it presently stands for optimum results wear a long sleeved shirt which is all one colour, and where the colour of the shirt is significantly different from the background.

4.  Detection of facial expressions needs improvement.  Presently the avatar only really displays left/right gaze direction, mouth open/closed and eyebrows up/down.  Detection of smiling has proven difficult, but probably could be achieved by looking for "laughter lines" either side of the mouth.  Detection of sad expressions is also quite tricky because the movements of the face can be quite subtle.  Any suggestions on these issues welcome.

5.  The Mr Plong avatar is quite crude, resembling something out of Southpark  With greater effort, more aesthetic avatars would be possible.  I havn't concentrated very much on the drawing/animation of the avatar, since this system was devised mainly for use on robots, with the avatar just acting as a crude means for me to determine whether the robot is understanding what it sees properly.

Webcams

When the program first runs it asks you to select a webcam from a big list of DirectShow devices.  It then creates a file called settings.txt in whatever is the current directory.  If you want to use a different camera just delete the settings.txt file and run the program again.

For best results use a USB2 or firewire webcam - preferably one which is based on a CCD chip rather than the more common CMOS type.  The CCD based cameras give a better quality image in low lighting conditions.  Image quality is especially important for determining the locations of facial features, such as eyebrows.

The program was designed assuming a default webcam image resolution of 320x240 pixels.  I've no idea what it would do if the default resolution were something other than this.

Portability

Although the present system runs only on Windows the DLL code which does all the calculations is just written in pretty standard C++, so it should be easily ported to other operating systems such as linux.

Possible future additions

1.  Detection of different styles of clothing.

2.  Detection of different hand/finger gestures (fist, open hand, pointing finger, etc)

Related Work

    EDISON

    Leonardo (MIT)

    Emotion Detection (Matthias Wimmer)

    A summary of human body language

    Robovie (Takayuki Kanda)